Tests are commonly performed when software is upgraded or changed to ensure that the software upgrade/change does not cause unexpected results. Many of these tests are historically performed manually and require significant time and expense. For instance, in the case of upgrading/changing software of a set-top box (STB), there can be thousands of test operations manually performed and verified, for each unique STB running that software. There is lot of dependency on manual verification of testing the uncompressed audio and video output of a device under test (DUT) such as a set-top box (STB), tablet, smartphones etc., which can be error prone. As a result, this may increase the test cycles significantly, especially for regression testing, impacting Time To Market requirements.
Example categories of testing performed for a STB include testing the power connection (power on/off, standby on/off), testing the High-Definition Multimedia Interface (HDMI) connection, testing commands to control the STB (e.g., Infrared commands, Bluetooth commands), testing audio and/or video quality, and testing the software to verify that it is operating as expected. These tests are typically manually intensive. For instance, testing the HDMI connection historically required someone to physically unplug and plug in the HDMI cable many times. As another example, some tests typically required a person physically watching a display connected to the STB in order to verify that the correct display and/or quality of display is being produced. Currently, there is no solution available to qualify a device under test (DUT) in an automated fashion using an outside-in approach that attempts to emulate human behavior.
Some testing solutions use a reference method where an input reference video stream is synchronized alongside the output of an AV endpoint and the two are compared. Such a solution has relatively high computation needs (e.g., synching the input reference video stream with the output of the AV endpoint), provisioning costs (e.g., setting up the sync of the input reference video stream and the output of the AV endpoint), frustration to the testers, and often requires the testers to watch the input reference video stream along with the output of the AV endpoint to compare the two.
Currently there is no way of continually monitoring the AV output of such devices to detect AV QoE issues using a no-reference (NR) method where the DUT are tuned to content through either a managed network or an unmanaged network at the end point of the video broadcast system where the end user experience is important.
In many present testing solutions that test live content, the DUT is tuned to the live content that is dynamic in nature, dependent on content provider, and may have errors embedded in it. Since the content itself may have issues, the test results may not give accurate assessment of the AV QoE produced by the DUT. Additionally, since the content may be copyrighted, it cannot be distributed to any external entity for STB qualification along with a test kit.